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Abstract. Suppose that / is a homomorphism from the mapping 
class group M.(N g ^ n ) of a nonorientable surface of genus g with n 
boundary components, to GL(m, C). We prove that if g > 5, 
n < 1 and m < g — 2, then / factors through the abelianization 
of M(N gin ), which is Z 2 x Z 2 for g £ {5,6} and Z 2 for g > 7. If 
g > 7, n = and m = <? — 1, then either / has finite image (of order 
at most two if g ^ 8), or it is conjugate to one of four "homological 
representations" . As an application we prove that for g > 5 and 
h < g, every homomorphism M.(N g> o) — > M(N h ^ ) factors through 
the abelianization of A4(N g ). 

1. Introduction 

For a compact surface F, its mapping class group Ai(F) is the group 
of isotopy classes of all, orientation preserving if F is orientable, home- 
omorphisms F — >■ F equal to the identity on the boundary of F. A 
compact surface of genus g with n boundary components will be de- 
noted by Sg tTl if it is orientable, or by N g<n if it is nonorientable. If 
n = then we drop it in the notation and write simply S g or N g . The 
first integral homology group of F will be denoted by H\(F). 

After fixing a basis of H\(S g ), the action of Ai(S g ) on H\(S g ) gives 
rise to a homomorphism Ai(S g ) — > Sp(2g,Z), which is well known to 
be surjective, and whose kernel is known as the Torelli group. Gluing 
a disc along each boundary component of Sg }Jl induces an epimorphism 
M(Sg tn ) -> M(S g ), and by composing it with M(S g ) ->■ Sp(2p, Z), 
and then with the inclusion Sp(2p, Z) GL(2p, C) we obtain the map 
$: A^(S , 9i „) — > GL(2g, C). Recently, the following two results were 
proved by J. Franks, M. Handel and M. Korkmaz. 

Theorem 1.1 ([6, 14]). Let g > 2, m < 2g - 1 and let f : M(S g , n ) -> 
GL(m, C) be a homomorphism. Then f is trivial if g > 3, and Im(/) 
is a quotient o/Zio if g = 2. 
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We say that two homomorphism fi, f 2 from a group G to a group 
H are conjugate if there exits h G H such that / 2 (:r) = hf\(x)h~ l for 
x e G. 

Theorem 1.2 ([15]). For g > 3, every nontrivial homomorphism 
/: A^S^n) — > GL(2g,C) conjugate to the map $. 

In this paper we prove analogous results for M(N g ). Fix g > 3. 
Let P ff denote the quotient of Hi(N g ) by its torsion. Hence, R g is 
a free Z-module of rank g — 1. There is covering P: S^-i — >■ iV 9 of 
degree two. By a theorem of Birman and Chillingworth [3], Ai(N g ) 
is isomorphic to the subgroup of M(S g -i) consisting of the isotopy 
classes of orientation preserving lifts of homeomorphisms of N g , which 
gives an action of N g on Hi(S g _i). Let K g C Hi(S g -i) be the kernel 
of the composition of the induced map P* : Hi(S g -i) — > H 1 (N y ) with 
the canonical projection H 1 (N g ) — > R g . Then K g is A^(iV 9 )-invariant 
subgroup of rank g — 1 and we have two homomorphisms 

tf i: .M(Ay ^GL(X 9 ) and M(N g ) G^H^Sg-^/Kg), 

which after fixing bases will be treated as representations of M(N g ) in 
GL(g — 1, C). We will see that these representations are not conjugate, 
although ker = ker ^>2- 

Our first result is the following. 

Theorem 1.3. Suppose thatn < 1, g > 5, m < g—2 and f : Ai(N g>n ) — > 
GL(m, C) is a nontrivial homomorphism. Then Im(/) is ether Z 2 or 
Z 2 x Z 2; the latter case being possible only for g = 5 or 6. 

Theorem 1.3 was proved in [14], in a more general setting of punc- 
tured surfaces, under additional assumption that m < g — 3 if g is even. 
Therefore, the only novelty of our result is that it also covers the case 
m = g — 2 for even g. As an application of Theorem 1.3 we prove the 
following result, which solves Problem 3.3 in [13]. 

Theorem 1.4. Suppose that g > 5, h < g and /: M(N g ) — > M(N h ) 
is a nontrivial homomorphism. Then Im(/) is as in Theorem 1.3. 

Analogous theorem for mapping class groups of orientable surfaces 
was proved in [10], see also [1]. We will prove that both Theorem 1.3 
and Theorem 1.4 fail for g = 4, by showing that there is a homomor- 
phism from M(N 4 ) to M(N 3 ) = GL(2, Z), whose image is isomorphic 
to the infinite dihedral group. 

Suppose that g > 7. Then the abelianization of Ai(N g ) is Z 2 and 
we denote by ab: M(N g ) — > Z 2 the canonical projection. For % — 1,2 
we set = (— l) ab ^j. Our next result is the following. 
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Theorem 1.5. Suppose that g > 7, g ^ 8 and /: M(N g ) ->■ GL(g - 
1,C) a nontrivial homomorphism. Then either Im(/) = Z 2 , or / 
conjugate to one of^i, ^2; ^2- 

For g = 8 other representations of .M (iV 8 ) in GL(7, C) occur, related 
to the fact that there is an epimorphism e: Ai(Ns) — > Sp(6,Z2) and 
the last group admits irreducible representations in GL(7, C) (see [24]). 
We prove the following result. 

Theorem 1.6. Suppose that /: A4(N 8 ) — > GL(7, C) is a homomor- 
phism. Then one of the following holds. 



To prove our theorems we use the ideas and results from [6, 14, 15] 
with necessary modifications. While the case of odd genus is relatively 
easy, the case of even genus requires much more effort. This phenom- 
enon is typical for the mapping group of a nonorientable surface. 

Throughout this paper we will often have to solve an equation of the 
form L = R, where L and R are products of matrices from GL(m, C) 
with some unknown coefficients. Although the dimension m is variable, 
the calculations of L and R always reduce to multiplication of blocks 
of size at most 7x7. With some patience, such calculations could be 
done by hand, but it is definitely easier to use a computer. We used 
GAP, but of course, any program that performs symbolic operations 
on matrices, could be used as well. 



Suppose that m > 2 is fixed. We denote by I m the identity matrix of 
dimension m. We will sometimes write simply /, if m is clear from the 
context. We denote by Eij the elementary matrix with 1 on the position 
and elsewhere. Suppose that M±, . . . , are nonsingular square 
matrices of dimensions mi, . . . , m&, where m\ + • • • + = m. Then 
we denote by diag (Mi, . . . , M k ) the mxm matrix with M ll . . . , M k on 
the main diagonal and zeros elsewhere. Set 



(1) Im(/) = Z 2 . 

(2) / or (-l) ab / factors through e: M(N 8 ) ->■ Sp(6,Z 2 ). 

(3) / is conjugate to one o/^i, ^2, 



2. Notation and algebraic preliminaries 
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For 2 < 2i < m we define 
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and for 2 < 2j < m — 2, 

Cj = diag (hj-2, W, I m -2-2j) ■ 

The proof of the following lemma is straightforward and we leave it as 
an exercise (c.f. [15, Lemma 2.2]). 

Lemma 2.1. Suppose that 1 < k < I < m/2 and M G GL(m,C) 
satisfies A^M = MA i} B { M = MB { and CjM = MCj for all i,j such 
that k < i < I, k < j < I — 1. Then M has the form 



for some A G C* ; where the top-left X of the block A/ 2 (z— fe+i) is at the 
position (2k — 1, 2k — 1) . 

Suppose that L G GL(m, C) and A is an eigenvalue of L. Then we 
denote by #A the multiplicity of A. For k > 1 we denote by E h (L, A) the 
space ker(E — XI) k . Thus E 1 ^, A) is the eigenspace of L with respect 
to A, and it will be also denoted by E(L, A). Note that if L' G GL(m, C) 
commutes with L, then the spaces E k (L, A) are L'-invariant for k > 1. 

For k > 2 we denote by &k the full symmetric group of the set 
{l,...,k}. It is generated by the transpositions <Tj = + 1) for 
1 < % < k — 1. We will need the following result from the representation 
theory of the symmetric group, see for example [7, Exercise 4.14]. 

Lemma 2.2. For k > 5, &k has no irreducible representation (over 
C) of dimension 1 < m < k — 1. If k >7, then &k has two irreducible 
representations of dimension k — 1: the standard one and the tensor 
product of the standard and sign representations. 

3. Mapping class group of a nonorientable surface 

Let n G {0, 1} and g > 2. Let us represent N g , n as a sphere (if n = 0) 
or a disc (if n = 1) with g crosscaps. This means that interiors of g small 
pairwise disjoint discs should be removed from the sphere/disc, and 
then antipodal points in each of the resulting boundary components 
should be identified. Let us arrange the crosscaps as shown on Figure 1 
and number them from 1 to g. For each nonempty subset / C {1, . . . , g} 
let £i be the simple closed curve shown on Figure 1. Note that is 
two-sided if and only if / has even number of elements. In such case 
t^ will be the Dehn twist about 7/ in the direction indicated by arrows 
on Figure 1. We will write £j instead of The following curves will 
play a special role and so we give them different names. 
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Figure 1. The surface N g>n and the curve for I = {ii,i2, ■ ■ ■ ,ik}- 




Figure 2. The crosscap transposition U{. 

• Si = £{i, i+ i} for 1 < i < g - 1, 

• £j = C{i,2,..,2j} for 2 < 2j < g. 
Note that e 1 = Si. 

For 1 < i < 3 - 1 we define the crosscap transposition Ui to be 
the isotopy class of the homeomorphism interchanging the i'th and the 
{i + l)'st crosscaps as shown on Figure 2, and equal to the identity 
outside a disc containing these crosscaps. 

The groups M.(Ni yn ) are trivial for n < 1 by [5, Theorem 3.4], we 
have M{N 2 ) = Z 2 x Z 2 by [16], and it follows from [3] that M(N 3 ) = 
GL(2,Z). For g > 3, a finite generating set for M.{N gtn ) was given in 
[4] for n = and [19] for n > 0. For n < 1 this set can be reduced to 
the one given in the following theorem, which can be deduced form the 
main result of [18]. 

Theorem 3.1. For g > 4 and n e {0,1}, M(N g ^ n ) is generated by 
u g _i, t £2 and t Si for 1 < i < g - 1. 

If n > 1, then we consider N 9jn as the result of gluing So,n+i to N 9t i 
along the boundary component. We will need the following relations, 
satisfied in Ai(N gtn ). Those between Dehn twists are the well know 
disjointness and braid relations. 

(Rl) t 5i t Sj = t Sj t Sl for \i -j\ > 1, 
(R2) t £i t £j = t e .t e . for alii, 
(R3) t £i t 5] = t 5j t £i for j ^ 2i, 



6 



BLAZEJ SZEPIETOWSKI 



(R4) t Si t Sl+1 t St = t Si+1 t Si t Si+ i for 1 < i < 9 - 2, 
(R5) t £i t S2l t £l = ts 2t t £ i S2l for 2% < g; 

The relations involving crosscap transpositions are not so well known 
and we refer the reader to [18] and [22] for their proofs. 

(R6) t Si Uj = Ujt Si for \i-j\>l, 

(R7) UiUj = UjUi for \i — j\ > 1, 

(R8) t £i Uj = Ujt £i for j >2i, 

(R9) UiU i+1 Ui = u i+1 UiU i+1 for 1 < i < g - 2, 
(RIO) t Si u i+ iUi = u i+1 Uit Si+1 for 1 < i < g - 2, 
(Rll) Ui + it 5l U l+1 Ui = t 5l t &l+1 for 1 < i < g - 2; 
(R12) t^Mit^ = Mi for 1 < i < g - 1. 

If follows from (R4) that all ts i are conjugate for 1 < i < g — 1, by (R5) 
t £j is conjugate to for 2j < g, and by (R12) is conjugate to tj 1 . 
Similarly, by (R9) all u,- L are conjugate for 1 < % < g — 1, and by (Rll) 
is conjugate to u^ 1 . 

For a group G we denote the abelianization G/[G,G] by G ab . The 
following theorem is proved in [12] for n = and generalised to n > 
in [19]. 

Theorem 3.2. For n < 1 and g > 3 ; «M(iV fljn ) ab /ias £/ie following 
presentation as a Z-module. 

([t Sl ], [t e2 ], [«i] | 2foJ = 2[t 62 ] = 2[ Ul ] =0) i/0 = 4, 
([^ 1 ],[«i]|2[ti 1 ] = 2[« 1 ] = 0> z/gG {3,5,6}, 
([«i]|2[ Ul ]=0) ifg>7. 

In particular, for g > 7 we have [t^J = 0. 

Lemma 3.3. For g > 5 and n < 1 let a, j3 be two-sided curves on 
N g ^ n , intersecting transversally in one point. If f: M.{N g ^ n ) — >■ G is 
a homomorphism, such that f(t a ) commutes with f{tp), then Im(/) is 
abelian. 

Proof. Let N = N g ^ n and Ai = M.(N g>n ). Fix a regular neighbour- 
hood A of a U j3. Note that A is homeomorphic to S^i and N\A is 
homeomorphic to N g _ 2 ,i- It follows that for each % < g — 2 there is a 
homeomorphism h: N — > N such that /i(a) = 5i and /i(/3) = It 
follows that htah^ 1 = t^ 1 and htph' 1 = t £ s 2 , where e { — 1,1} for 
j — 1,2. Hence /(^) commutes with f(t$ i+1 ) and by the braid relation 
(R4) /fo) = /(^ i+ J. Analogously, /(* ea ) = /(t* 4 ). By Theorem 3.1, 
Im(/) is generated by fitsj and f(u g -i), and since M g _! commutes 
with tin thus Im(/) is abelian. □ 
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Figure 3. The surface S g _i for g = 2r + 1 (top) and 
g = 2r + 2 (bottom). 

Lemma 3.4. Suppose that g > 4 and f : A4(N g!n ) G is a homomor- 
phism. If f(t e .) = f(t s .) for some 2% + 1 < j < g - I, then f(t 2 Si ) = 1. 

Proof. Set x = /(i £i ) = /(t^.) and y = f(uj). By the relation (R8) we 
have xy = yx, and by (R12) xyx = y. Hence x 2 = 1 which finishes the 
proof, because t$. is conjugate to ts 1 - □ 

Let g = 2r + s, where r > 1, s e {1,2} and 5 = S'g-i. Consider 
5 as being embedded in M 3 in such a way that it is invariant under 
the reflections about the xy, xz and yz planes, as shown on Figure 3. 
We define a homeomorphism j: S — > S as j(x,y,z) = (—x,—y,—z). 
The quotient space S/j is a nonorientable surface of genus g and the 
projection p: S — >■ S/j is a covering map of degree 2. Let S" be the 
subsurface of S consisting of points (x, y,z) G S with x < —e, where 
e is a positive constant, so small that S' is homeomorphic to SV,s- If 
g is even, then one of the boundary components of S' is isotopic to 
a r+ i. In this paper we identify isotopic curves, and therefore we will 
treat a r+ i as a curve on S'. Note that the restriction of p to S' is 
an embedding. For odd g we define 7' to be the arc of 7 r consisting 
of points with x < 0. For even g we define to be the arc of (3 r+ \ 
consisting of points with x < 0. Note that ^(7') and p(j3') are one-sided 
simple closed curves on S/j. 
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Proposition 3.5. There is a homeomorphism <p: S g -\/j — > N g such 
that, for P = if o p, up to isotopy 



(1) P(A) = 5 2l for\<i<r, 

(2) P( ai ) = e % for 2<2%<g, 

(3) P( 7i ) = 5 2t+1 for2<2i<g-2, 

(4) Ptf) = £ g if g is odd, 

(5) p(p') = £ g if g is even. 



Proof. Observe that the curves 5i for 1 < i < g — 1 form a chain of 
two-sided curves, which means that 5i and 5j intersect at one point if 
\i — j\ = 1, and they are disjoint otherwise. It follows that a regular 
neighbourhood of the union of 5i for l<i<g — lis homeomorphic to 
SV,s- Let £ be such a neighbourhood, which may be taken to contain 
the curves £j for 2 < 2i < g (if g is even, then one of the boundary 
components of E is isotopic to e r+ \). Note that e i+ i and S 2 i+i 
bound a pair of pants for 2 < 2i < g — 2. It follows that there exists 
a homeomorphism </?: S g ^i/j — >■ iV g such that, for P = </? o p, we have 
P(S') = S and the conditions (1, 2, 3) are satisfied. Observe that N g \E 
is a Mobius strip (if g is odd) or an annulus (if g is even), whose core 
(isotopic to £{i,..., 9 }) intersects £ 9 once. By looking at the intersection 
of £ g with the curves 5 iy Sj it is easy to see that ip can be taken to 
satisfy also the condition (4) or (5). □ 

Corollary 3.6. There is a homomorphism t: M(S') — > M(N g ^ n ) such 
that 



where the Dehn twists about the curves on S' are right with respect to 
the standard orientation. 

Proof. By the proof of Proposition 3.5, the restriction of P to S' is 
a homeomorphism onto £ satisfying the conditions (1,2,3). There is 
an induced isomorphism Ai(S') — > which may be composed 

with the homomorphism A^(S) — > M.{N g ^ n ) induced by the inclusion 
E N gtn , for any n > 0, to obtain i. □ 

For any homeomorphism h: N g — >■ N g there is a unique orientation 
preserving lift h: S g -\ — > S g -\ such that h o P = P o h. By [3], the 
mapping h h induces a monomorphism 6: Ai(N g ) — > A4(S g -i). The 
following proposition follows from [3] and [22, Theorem 10], where the 
lift of a crosscap transposition is determined. 



i«5 2l for\<i <r, 
t £i for 2<2i<g, 
Cn /or 2 < 2 ? < (? - 2 ; 



LINEAR REPRESENTATIONS OF THE MAPPING CLASS GROUP 9 

Proposition 3.7. There is a monomorphism 9: A4(N g ) — > Ai(S g -i) 
such that 

0(tei) = t ai tal_i, d{t S2i ) = tfrtp^., 0(t 52j+1 ) = t^t'^^., 

for l<i<r,2<2j<g — 2 and 

e(u )= feWM/A+i) 2 ^ 1 ifg = 2r + l, 
9 l^X+i^+iM^+JV ifg = 2r + 2. 

4. HOMOLOGICAL REPRESENTATIONS 

Fix g > 3 and let S = S g ^, N = N g and P: S N be as in 
the previous section. The group Hi(S) is a free Z-module of rank 
2{g — 1) and the homology classes a« = [a*], hi = [fy] for 1 < % < g — 1 
form its basis, which is a symplectic basis with respect to the algebraic 
intersection form: 

(di, cij) = 0, (bi, bj) = 0, (di, bj) = 5ij. 

Let $: Ai(S) — >■ Sp(i/i(S')) be the homomorphism induced by the 
action of M(S) on Hi(S). If 7 is an oriented simple closed curve on S, 
[7] G Hi(S) is its homology class, and t 7 is the right Dehn twist, then 
$(t 7 ) is the transvection 

(4.1) *(ty)(h) = h+ ([7], fc> [7], for h G ^(5). 

From this formula we immediately obtain that, with respect to the 
basis (ai, 61, ... , a ff _i, 6 9 -i), we have 

®(t at ) = A, $(t ft ) = P 4 , $(f 7j .) = c,-, 

for 1 < i < g — 1, 1 < j < g — 2, where Aj, Pj and Cj are the matrices 
defined in Section 2. 

The group Hi(N) has the following presentation, as a Z-module: 

Hi(N) = ( Xl ,...,x g \ 2(xi + • • • + = 0) , 

where X{ = [&]. Set A; = xi + • • • + x g and P = H\(N)/ (k). Observe 
that k is the unique element of order two in Hi(N) and R is a free 
Z-module of rank g — 1. 

The map P: S ^ N induces P*: Pti(S) ->■ Pi(AQ, such that, for 
1 < i < r 

P*(oj) = 27 H Yx 2i = -P*(a ff _j), 

P*(6i) = x 2i + z 2 i+i = P*(b g -i), 
and if <? = 2r + 2 then 

P*(ar+i) = a:i H \- x g = k, P*(6 r+ i) = 2x r 
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Let q: Hi(S) — > R be the composition of P* with the canonical projec- 
tion Hi(N) — >■ i?, and set K = kerg. It is easy to verify that K has 
rank g — 1 and the following elements form its basis: 

€i = a,i + a g _i, e r+i = h- b g _i for 1 < i < r, 
e 2r+ i = a r+1 for g = 2r + 2. 

We also set 

fi = hi, fr+i = a g _i for 1 < % < r, 

f 2r +i = b r+1 for g = 2r + 2. 

Observe that the elements e i: fi for 1 < % < g — 1 form a symplectic 
basis of Hi(S). It follows that Hi(S)/K is a free Z- module of rank 
g — 1, which is canonically isomorphic to if (7 is odd, or to an index- 
two subgroup of R if g is even. The group Ai(N) acts on Hi(S) by the 
composition $ o 9: M(N) ->■ Sp(#i(S)). Observe that X is M(N)- 
invariant and hence we have two (g — l)-dimensional representations 

^1: M(iV) _). GL(if), ^ 2 : M(N) ->• GL(// 1 ( ( S)/X). 

Lemma 4.1. ker^i = ker^ 2 and 0(ker\l/i) C ker$. 

Proof. Fix the basis (ei, . . . , e g _i, fi, . . . , f g -i) of Hi(S). For any x G 

M(N) let X be the matrix of $(6(x)). We have X = ^ ^ , where 

Xi,X 2 ,Y are (</ — 1) x (g — 1) matrices. The matrix of the algebraic 

intersection form is fi = ^ j ^Q^j an< ^ s i nce X is symplectic, we 

have X l ttX = fi, which gives X\X 2 = I. Therefore X x = I X 2 = I, 
which proves ker^i = ker^ 2 . To prove the second part of the lemma, 
assume X\ = X 2 = I . Let j* : Ai(S) — > Ai(S) be the map induced by 
the covering involution j. It is easy to check that the matrix of j* has 

the form J = ( ) for some T. We have XJ = JX, which 

V u VV 

implies Y — 0. □ 

Note that ker <3> is the Torelli group, which is well known to be tor- 
sion free, and since 9 is a monomorphism, we immediately obtain the 
following. 

Corollary 4.2. ker ^1 is torsion free. □ 

Remark 4.3. Let H denote the subgroup of A4(N) consisting of the 
elements inducing the identity on Hi(N). It was proved in [9] that 
9(H) C ker $. We leave it as an exercise to check that if g is odd, then 
H = ker\l>2, whereas if g is even, then H is an index-two subgroup 
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of ker\l/2- In the latter case, if g = 2r + 2, then we have ker ^2 = 
HUt e ,.H. 

Remark 4.4. There is a nontrivial action of ~K\(N) on Z defined as 
follows: 7 G tx\{N) acts by multiplication by 1 or —1 according to 
whether 7 preserves or reverses local orientations of N. This action 
gives rise to homology groups with local coefficients H*(N, Z), where Z 
is Z with the nontrivial Z[7r 1 (A^)]-module structure. By [11, Example 
3H.3], we have the exact sequence 

H 2 (N) -)• #i(iV,Z) -> if^S) A Pf^iV), 

which is a part of a long exact sequence of homology groups. Since 
Hz{N) = 0, we have a Ai (iV)-equivariant isomorphism H\(N, Z) = 
kerP*. If 5 is odd, then kerP* = K, whereas if g is even, then kerP* 
is an index-two subgroup of K. Therefore the representations \&i and 
^2 m ay be seen as coming from the actions of Ai(N) on Hi(N, Z) and 
Hi(N) respectively. 

For K we fix the basis 

(ei,e r+ i, . . . ,e r ,e 2r ) ify = 2r + l, 
(ei, e r+ i, . . . , e r , e 2r , e 2r +i) if g = 2r + 2. 
For H 1 (S)/K we fix the basis 

(ai + X, 61 + if, . . . , a r + K, b r + K) if g = 2r + 1, 

(ai + K, 61 + X, . . . , a r + K, b r + K, b r+l + K) if g = 2r + 2. 

Having fixed bases for K and Hi (S) /K we can now compute, for ^1 and 
^2, the images of the generators of Ai(N). This is done by a straight- 
forward calculation, using Proposition 3.7 and the formula (4.1). For 
k = 1, 2 and l<i<r, l<j<r — lwe have 



If = 2r + 1 then 



A, m k {U 2i )=B l , m k (ts 2l+1 ) = C : 





1 
1 





1 
1 








*2(V.) = 



'9-3 






/V4 




V 



N 




\ 

1 1 -2 
1 

1 / 
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/%_ 4 oo o \ 

1-11 

10 

\ 2 -I J 



// 9 _4 oo o \ 

11-2 

10 

\ 1-1/ 



Now it is easy to see that and \I/ 2 are not conjugate as homomor- 
phism to GL(p — 1, C). For suppose that there is M G GL(g — 1, C), 
such that = M^ 2 (x)M~ 1 for all x G A4(iV)- Then M com- 

mutes with Aj, Bi, Cj ;for 1 < i < r, 1 < j < r — 1, and by Lemma 2.1, 
M = a/ 2r if 5 = 2r+l, or M = diag(a/ 2r , 0) if g = 2r + 2, for G C. 
In either case it is impossible that = M^ 2 (%-i)-^ _1 - 

5. HOMOMORPHISMS FROM M(Ng jTl ) TO GL(m, C) FOR m < g -1 

The aim of this section is to prove Theorem 1.3. The proof is divided 
in two parts. 

Proof of Theorem 1.3 for (g, m) ^ (6, 4). Suppose that n G {0, 1}, g = 
2r + s for r > 2, s G {1,2}, m < g - 2 and /: -M(iV ff , n ) ->■ GL(m, C) 
is a homomorphism. By Theorem 3.2, it suffices to prove that Im(/) is 
abelian. Let S' = S rjS and 1: Ai(S') — > A4(N gjJl ) be the homomorphism 
from Corollary 3.6. Set f' — fou and observe that if Im(/') is abelian, 
then so is Im(/), by Lemma 3.3. 

Suppose that m < 2r — 1. Then Im(/') is either trivial or cyclic by 
Theorem 1.1 and we are done. This finishes the proof for odd g. 

Suppose that g = 2r + 2 for r > 3 and m = 2r. By Theorem 1.2, /' is 
either trivial or conjugate to the homological representation $. In the 
former case we are done. In the latter case, by the definition of <3> we 
have $(t 7r ) = &(t ar ) because the curves 7 r and a r become isotopic after 
gluing discs to the boundary of S'. It follows that f{ts 2r+1 ) — f(te r ) 
and by Lemma 3.4 f(tg ) = 1. This is a contradiction because $(t ai ) 
has infinite order. □ 

In order to prove Theorem 1.3 for (g, m) = (6, 4), we first prove some 
lemmas. 

Lemma 5.1. Suppose that f: A^(A^ 4 n ) — > GL(2, C) is a homomor- 
phism. Then, with respect to some basis one of the following cases 
holds. 

(1) f(t Sl ) = f(t 52 ) = f^) = XI, X G {-1, 1} 

(2) f(t Sl ) = f(t S2 ) = f(ts 3 ) ~- n ° 



-1 

Q I > f (^2 ) \ j 1 



(3) f(t Sl ) = f(t S3 ) = ( I \ ' ° 
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In particular f(tg) = 1. 

Proof. For % = 1, 2, 3 let = /(ij.) and [/ = /(u 3 ). 

Suppose that L\ has only one eigenvalue A. Since Li is conjugate to 
L^ 1 (by (R12)), we have A G {-1,1}. If dim£'(L 1 , A) = 2, then we have 
the case (1). Suppose that dim E(L U A) = I. If A) ^ E(L 2 ,X), 

then with respect to some basis we have L\ — ( ^ ^ ), L 



\J' z \x A 
for some x, and from the braid relation L1L2L1 = L 2 L\L 2 we have 

x = —1. Since L 3 commutes with Li we have L 3 = I ^ ^ J for some 



A y 

y, and from L 2 L 3 L 2 = L 3 L 2 L 3 we obtain y — 1, hence L 1 — L 3 . Since 
<5i = £1, we have = / by Lemma 3.4 (for i = 1, j = 3), which 
is a contradiction. If E(L 1 ,X) = E(L 2 ,X), then with respect to some 

A A r fX x 



basis we have L\ = y^ji L 2 = ^ J , and it is easy to obtain a 

contradiction as above, by showing that L\ = L 2 = L 3 . 

Suppose that Li has two eigenvalues A, fi. Then with respect to some 

basis we have L\ = > an< ^ smce -^3 an d ^ commute with L 1 , they 

are also diagonal. In particular we have UL 3 = L 3 U and L 3 UL 3 = U 
(R12) gives L\ = 1, which implies {A,/i} = { — 1, 1}. We have L 3 = L x 
or L 3 = —L\. In the latter case the braid relations L 3 L 2 L 3 = L 2 L 3 L 2 
and L\L 2 L\ = L 2 L±L 2 imply L 2 L±L 2 = 0, a contradiction, hence L\ = 
L 3 . 

If E(Li, 1) 7^ E(L 2 , 1), then with respect to some basis we have L\ = 

From L\L 2 L\ = L 2 L\L 2 we have x — 1 and 



-y z ^ x i y 

we are in the case (3). Analogously, if E(L\,—\) 7^ E(L 2 ,—1), then 
with respect to some basis we have L 1 — ( J~ j ) , L 2 



x \y z V 1 _1 , 
and since E(L 1 , 1) 7^ E(L 1 , 1), we are in the case (3) again. 

Finally, if 1) = E{L 2 ,1) and £'(L 1 , -1) = E{L 2 ,-1), then 

with respect to some basis we have L 1 = L 2 = ( „ ) and we are in 



the case (2). □ 

Lemma 5.2. Suppose that n < 1 and /: -M(iV 6)n ) — >■ GL(4,C) is a 
homomorphism such that f(tg ) = 1. T/zen Im(/) zs abelian. 

Proof. Let if be the normal closure of in -M(iV 6)n ) and set G = 
M(N &)n )/H. We have an induced homomorphism /': G — > GL(4, C) 
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such that / = /' o -a, where n: M(N &>n ) — > G is the canonical projec- 
tion. By the relations (Rl, R4), the mapping p(o~j) = ^(^J, where <7j 
is the transposition (i,i + 1) for 1 < % < 5, defines a homomorphism 
p: ©6 — ^ C. Let 0: ©6 "~ > GL(4, C) be the composition /' o p. By 
Lemma 2.2, is the direct sum of one-dimensional representations. In 
particular the image of is abelian, and so is Im(/) by Lemma 3.3. □ 

Let R be the subsurface obtained by removing from N^ n a regular 
neighbourhood of 5\ U5 2 . Note that R is homeomorphic to N 4jJl+1 . The 
homomorphism A4(R) — > A4(N 6jn ) induced by the inclusion of R in 
N 6tn is injective, and we will treat A4(R) as a subgroup of A4(N 6tn ). 

Lemma 5.3. Suppose that n < 1, f: Ai(N 6tTl ) — > GL(4, C) is a ho- 
momorphism and there exists a splitting C 4 = V\ © Vi such that Vi is 
a 2 -dimensional A4(R) -invariant subspace for i — 1, 2. Then Im(/) is 
abelian. 

Proof. Let /' be the restriction of / to M(R). With respect to the 
splitting C 4 = Vi © V 2 we have f' = f l ® f 2 for some f { : M(R) ->■ 
GL(2,C), i = 1,2. By Lemma 5.1 we have fi{t 2 s J = 1 for i = 1,2, 
hence f(tg ) = 1 and we are done by Lemma 5.2. □ 

Lemma 5.4. Suppose that n < 1, f: M{N 6tn ) — >■ GL(4, C) is a 
homomorphism, /(tjj /ias on/y one eigenvalue and there exists a 2- 
dimensional M(R) -invariant subspace. Then Imf is abelian. 

Proof. Let A be the eigenvalue of f(t Sl ). Fix a basis of C 4 whose first 
two vectors span the M. (-R)-invariant subspace. By the case (1) of 

Lemma 5.1, with respect to such basis we have f(ts 4 ) = 

f(ts 5 ) = ^j^j , for some 2x2 matrices X, Y. In particular f(tg 4 ) 

and f(ts 5 ) commute and we are done by Lemma 3.3. □ 

Proof of Theorem 1.3 for g = Q, m = 4. Suppose that n G {0, 1} and 
/: At(A^,n) — > GL(4,C) is a homomorphism. For 1 < % < 5 we set 
Li = f(tsi) and M = f(t £2 ), U 5 = f(u 5 ). We consider the following 
cases. 

(1) Li has 4 eigenvalues. 

(2) Li has 3 eigenvalues. 

(3) Li has 2 eigenvalues with equal multiplicities. 

(4) L\ has 2 eigenvalues with different multiplicities. 

(5) L\ has 1 eigenvalue. 

In the cases (1, 2, 3) it is easy to find a splitting C 4 = V\ © V 2 such 
that Vi is a 2-dimensional M (i?)-invariant subspace for % — 1,2. For 
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example, suppose that L\ has three eigenvalues Ai,A2,A3 such that 
#Ai = #A 2 = 1 and #A 3 = 2. Then we take V x = £(Li,Ai) © 
E(L U A 2 ) and \/ 2 = £'(L 1 , A 3 ) if dim E(L U A 3 ) = 2 or V 2 = E 2 (L ± , A 3 ) 
if dim£ , (L 1 ,A 3 ) = 1. Therefore in the cases (1, 2, 3) we are done by 
Lemma 5.3. 

Assume (5). Let A be the unique eigenvalue of L\ and k = dim E(L±, A). 
If k — 4 then Li = XI and the image of / is cyclic. If k — 2 
or /c = 1 then respectively E(Li,X) or E 2 (Li,\) is a 2-dimensional 
A^(i?)- invariant subspace, and we are done by Lemma 5.4. Suppose 
that k = 3. If £(Li, A) ^ £(L 2 , A) then A) n £(L 2 , A) is a 2- 

dimensional A4(R)- invariant subspace, and we are done by Lemma 5.4. 
If E(Li, A) = E(L 2 , A) then with respect to some basis we have 



(\ o o\ 

A 

A 1 

\o o o Ay 



A o o i\ 

A y 

A z 

\0 XJ 



In particular L\ and L 2 commute and we are done by Lemma 3.3. 

It remains to consider the case (4). Suppose that L\ has eigenvalues 
/i, A, with #// = 1 and #A = 3. Since Li is conjugate to Lj" 1 , we 
have {/i, A} = { — 1,1}. It follows from Theorem 3.2 that there is 
a homomorphism t(A4(Nq)) — > { — 1,1} such that r(ai) = —1. By 
multiplying / by r if necessary, we may assume fx = — 1, A = 1. The 
Jordan form of Li is one of the following three matrices. 



(0 



/-l o o\ 

10 

10 

\ 1/ 



/-l o o\ 

110 

11 

y 1J 



(in) 



f-1 o o\ 

10 

11 

\ 1J 



In the case (i) we have L\ — I and we are done by Lemma 5.2. 
In the case (ii) the following subspaces are A4(-R)-invariant: E(L 1: —1), 
E(L U 1), E 2 {L U 1), E 3 {L U 1). It follows that 



M = 



(x x 0\ 

x 2 vi v 2 

x 3 v 3 

\0 x A j 



La 



/yi \ 

y 2 iui w 2 

y 3 w 3 

\o o o 2/ 4 y 



The braid relation ML 4 M = L 4 ML 4 (R5) implies Xj = for 1 < « < 4. 
Since the first two vectors of the basis are eigenvectors of M, they have 
to correspond to different eigenvalues of M. Therefore x 2 = —x±, 
x 3 = X4 = 1 and x± — 1 or x\ — — 1. In either case it is not difficult 
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to check that ML 4 M = L4ML4 holds if and only if M = L 4 . We are 
done by Lemma 3.3. 

In the case (iii) the following subspaces are M. (i?)-invariant: E(Li, — 1), 
E(Li, 1), E 2 (Li, 1). We have dim E(L\, 1) = 2 and by applying Lemma 
5.1 to the action of M.{R) on this subspace, we obtain three sub-cases. 

Sub-case (iiia). If the action of M(R) on E(Li, 1) is trivial, then we 
have 



M = 



/-l 







1 





U = 



(-1 






1 





o\ 

Vi 

V2 



As in the case (ii), the braid relation implies M = L 4 and we are done 
by Lemma 3.3. 

Sub-case (iiib). By changing the basis of E(Li,l) we may assume 
that 



M 



(\ 







-1 







1 




Xi 



(\ 








-1 






V2 



1 



As in the case (ii), the braid relation implies M = L 4 and we are done 
by Lemma 3.3. 

Sub-case (iiic). By changing the basis of E(L 1: 1) we may assume 
that 



M 



(\ 








1 

-1 





Xi 



/I 








-1 
1 





yi 



/1 








1 

-1 




V 



By solving the equations ML4M 
we obtain x 2 = — (2xi + yi + 2y 2 ) 
ML 5 = L 5 M we obtain a; 2 = z 2 . 



= and L5L4L5 = L4L5L4 

z 2 = -(2^i + yi + 2y 2 ), and from 
Thus M = L 5 and by Lemma 3.4 

□ 



L{ — \. We are done by Lemma 5.2. 

6. HOMOMORPHISMS BETWEEN MAPPING CLASS GROUPS 

The aim of this section is to prove Theorem 1.4. Fix g > 5 and set 
= M.(N g ). We are going to use the fact that s — ts 1 • • ~ts g _ 1 has 
finite order in M. (equal to g if it is even, or 2g otherwise, see [18]). By 
the relations (R1,R4) we have 

(6.1) t Si+1 s = st Sl for 1 < 2 < # - 2. 

By Theorem 3.2 we have s E [M, M] for g > 7 and g = 5, s 2 E [M,M] 
for g = 6. 
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Proof of Theorem 1.4- Suppose that g > 5, h < g and /: A4(N g ) — > 
Ai(N h ) is a homomorphism. Since M(N h ) is abelian for h < 2, we are 
assuming h > 3. 

Let /': M(N g ) ->• GL(h - 1, C) be the composition 5^/ and if = 
ker^x. By Theorem 1.3, Im(/') is abelian, hence f([M(N g ), M(N g )}) C 
if. Suppose that g > 7 or g = 5. Then /(s) G if, and since if is torsion 
free by Lemma 4.2, thus f(s) = 1. This gives, by (6.1), /(^J = /(^J 
and we are done by Lemma 3.3. If g — 6 then /(s 2 ) G if, which gives 
/(s 2 ) = 1 and f(ts 2 ) = f(ts 4 )- Since ts 1 commutes with t$ 4 , thus /(t^J 
commutes with /(t«5 2 ) and we are done by Lemma 3.3. □ 

Note that Theorems 1.3 and 1.4 are trivially true for g < 3 because 
GL(1, C) = C*, M(N 2 ) = Z 2 x Z 2 , M(iVi) = 1 are abelian groups. On 
the other hand, Corollary 6.2 below shows that both theorems are false 
for g = 4 (recall that A^(A^ 3 ) = GL(2,Z)). Let denote the infinite 
dihedral group, defined by the presentation 

Doo = (x,y \x 2 = y 2 = l) . 

Lemma 6.1. There is an epimorphism 0: Ai(N^) — > D^. 

Proof. According to the main result of [21] simplified in [18], A^(A^) 
admits a presentation with generators t £2 , t^, Ui for i = 1,2,3 and 
relations (Rl, R3, R4, R6, R7, R9, RIO, Rll, R12) and 

t Si+1 u 1 u i+1 = UiU i+1 t Si for i = 1,2 

(t £2 u 3 ) 2 = 1, ts 1 (t S , 2 ts. i U 3 U 2 )ts 1 = (ts 2 ts 3 U3U 2 ). 

It is easy to check that the mapping 4>{t £2 ) = xy, (pitsj = 1, 4>{uj) = y 
for i = 1,2, 3, respects the defining relations of ^(A^), hence it defines 
a homomorphism onto D^. □ 

Corollary 6.2. For h > 3 there is a homomorphism f: A^A^) — > 
Ai(N h ), such that Im(/) is isomorphic to D^. 

Proof. Fix h > 3. By the proof of [20, Theorem 3], t$ 1 can be written in 
Ai(Nh) as a product of two involutions cr, r. Since t$ 1 has infinite order 
in A4(N h ), the mapping x i— > a, y i— )■ r defines an embedding — > 
Ai(N h ). By pre-composing this embedding with the epimorphism 
from Lemma 6.1, we obtain /. □ 

The following two theorems can be proved by the same method as 
Theorem 1.4. We leave the details to the reader. 

Theorem 6.3. Suppose that g > 5, g > 2h + 2 and f : M(N g ) ->• 
Ai(Sh) is a homomorphism. Then Im(/) is abelian. 
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Theorem 6.4. Suppose that g > 3 and h < 2g. Then the only homo- 
morphism from A4(S g ) to A4(N h ) is the trivial one. 

7. HOMOMORPHISMS FROM M(N g ) TO GL(g - 1,C). 

The aim of this section is to prove Theorem 1.5. The prove it divided 
in two cases, according to the parity of the genus. 

Let g = 2r + s, s G {1,2}, S' = S r>s and t: M(S') ->■ M(N g>n ) be 
the homomorphism from Corollary 3.6. If /: M(N g ^ n ) — > GL(m, C) is 
a homomorphism, then we set /' = / o i. 

Proof of Theorem 1.5 for odd g. Suppose that N = N 2r+1 , r > 3 and 
/: M(N) — > GL(2r, C) is a homomorphism, such that Im(/) is not 
abelian. By Theorem 1.2, /' is conjugate to the homological represen- 
tation $, and thus the exists a basis, such that f(t 6i ) = f'(t ai ) = Ai, 
/(**«) = f'M =B l iovl<i<r and f(t d2j+1 ) = /'(^) = Cj for 
1 < j < r - 1. Set C/fc = /(u fc ) for 1 < fc < 2r. 

Since \Ji r commutes with Ai and Bi for 1 < % < r, and with Cj for 
j = 1 , . . . , r — 2 (R6,R8) thus, by Lemma 2.1, 

u (Xhr-2 

U2r ~ { X 

for some 2x2 matrix X. Since f/ 2r is conjugate to U^ 1 we have A G 
{— 1, 1} and by multiplying / by (— l) ab if necessary, we may assume 

A = 1. The relation B r U 2r B r = U 2r (R12) implies X = . 

From (Rll) and (R7) we have 

U 2 r-2 = (C r -iB r B r -iC r -i) 1 U 2r {C r -\B r B r -\C r -\)^ 
U 2r U 2r ^ 2 — U 2r - 2 U 2r = 0, 

and since the left hand side of the last equation is equal to 

(1 — X 2 )(E 2rt2r ^3 + E 2r - 2j2 r-l), 

thus x 2 = 1. We have U 2r l = U 2r , and from (Rll) and (R9) 
U 2r -i = (C r -iB r ) 1 U 2r {C r -iB r ), 
U 2r U 2r ~\U 2r — U 2r -iU 2r U 2r -i — o. 

By considering the 1 and x — — 1 separately, we find that 

the left hand side of the last equation is of the form (y — x) 2 Z, where 
Z 7^ 0. Hence x = y and U 2r = ^i(u 2r ) if x — 1, or U 2r = ^ 2 (u 2r ) 
if x = —1. By Theorem 3.1, / is equal to \&i or ^2 on generators of 
M(N). □ 
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Now we will borrow some arguments from [15] to prove Lemma 7.3 
below, which will be a starting point for the proof of Theorem 1.5 for 
even genus. 

Lemma 7.1. Suppose that n < 1, g > 5 and f : M(N g ^ n ) — > GL(m, C) 
is a homomorphism. If there is a flag = Wq C W\ C • • • C Wk = C m 
of M.(N g n ) -invariant subspaces such that dim(W / j/W / j_i) < g — 1 /or 
i = 1, . . . ,k, then Lm(/) is abelian. 

Proof. The same argument as in the proof of [15, Lemma 4.8] can be 
applied, using Theorem 1.3, to show that with respect to some basis 
f[M.{N g>n ) ) Ai{N g>n )\ is contained in the subgroup of upper triangular 
matrices with 1 on the diagonal. Since this subgroup is nilpotent and 
[M(S'),M(S')] is perfect, it follows that f'[M(S'),M(S')] is trivial, 
which means that Im(/') is abelian, and so is Im(/). □ 

Lemma 7.2. Suppose that N = N 2r +2, r > 3 and f : M(N) ->■ 
GL(2r + 1,C) is a homomorphism, such that Im(/) is not abelian. 
Then L 1 = /(^J has an eigenvalue A such that dimi?(L 1 , A) = 2r. 

Proof. By [15, Corollary 4.6] applied to /', L\ has at most two eigen- 
values. It follows that there is an eigenvalue A with #A > r + 1 > 4. 
Set m = dim£'(Li,A). Since Im(/) is not abelian, thus m < 2r. We 
are going to show that m = 2r. 

Let R be the subsurface obtained by removing from iV a regular 
neighbourhood of o\ U S 2 . We have R w N 2r ,i- We treat M(R) as a 
subgroup of A4(N). 

Suppose m < 2r — 2. Let W = E k (L 1 , A), where k = max{4 — m, 1}. 
Observe that W is a M. (i?)-invariant subspace with 3 < dim W < 2r — 
2. By Lemma 7.1, f(A4(R)) is abelian, which means f(ts 4 ) = f(ts B )- 
By Lemma 3.3, Im(/) is abelian, a contradiction. 

Suppose that m = 2r — 1 and set L 2 = f(ts 2 )- If E(Li, A) ^ E(L 2 , A) 
then E(Li,X) D E(L 2 ,\) is a (-R)-invariant subspace of dimension 
2r — 3 or 2r — 2 and we can use the same argument as above to obtain a 
contradiction. If E(L l7 A) = E(L 2 , A), then by [15, Lemma 4.3] applied 
to /', E(L 1 ,X) is a A^(S")-invariant subspace of dimension 2r — 1, 
and by [15, Lemma 4.8] /' is trivial. It follows that Imf is abelian, a 
contradiction. □ 

Lemma 7.3. Suppose that N = N 2r+2 , r > 3 and /: M(N) ->■ 
GL(2r + 1, C) a homomorphism. If r = 3 £/ien assume that 1 zs 
i/ie unique eigenvalue of fit T/ien either Im(/) zs abelian, or with 
respect to some basis f(t £j ) = A i} f(ts 2i ) = P>i for i — 1, . . . , r. 

Proof. Suppose that Im(/) is not abelian. By Lemma 7.2, Li = /(^J 
has an eigenvalue A with dim£'(L 1 ,A) — 2r. If r = 3 then A = 1 
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by assumption, and for r > 4, A = 1 by the proof of [15, Lemma 
5.2]. Since M(S') is perfect, thus detLi = 1 and A = 1 is the unique 
eigenvalue. Set L 2 = f(ts 2 )- We claim that E(L 1 ,1) ^ E(L 2 ,l). For 
otherwise it is easy to prove that L\ and L 2 commute (see the proof 
of Theorem 1.3 for (g,m) = (6,4), case (5)), and Im(/) is abelian by 
Lemma 3.3, a contradiction. Now we can apply [15, Lemma 4.7] to /' to 
conclude that with respect to some basis we have f(t £ .) = f'(t a .) = 

= f'(t lk )=B i for i = l,...,r. □ 

Proof of Theorem 1.5 for even g. Suppose that iV = N 2r+2 , r > 4 and 
/: Ai(N) — > GL(2r + 1,C) is a homomorphism, such that Im(/) is 
not abelian. By Lemma 7.3 there is a basis such that f(t Ei ) = Aj and 
f(ts 2i ) = Bi for 1 < i < r. Set Di = f(t$ 2i+1 ) for 1 < i < r and 
Uj = f(uj) for 1 < j < 2r + 1. 

Fix % G {1, ...,r — 1}. Since Di is conjugate to Ai, it has one 
eigenvalue A = 1. For j ^ {i,i + 1} the relations D{Aj = AjDi and 
DiBj = BjDi imply, by Lemma 2.1, that Di has the form 



/ h(i-i) 











o\ 





F u 


F\2 





x 1 





F 2 i 


F22 





x 2 











h(g-i-l) 





V 


Y 1 


Y 2 








where F k i are 2x2 matrices, X k are 2x1 vectors, Y\ are 1x2 vectors 
and z is a complex number. The relations D^Ai = AiDi and DiA i+ i = 
A l+1 Di imply, for k, I G {1,2}, VF kl = F kl V, VF kl = F kl for k ^ I, 
VX k = X k , Y t V = Y h hence 

Fll ~{o Sl )> Fl2 -{o oJ' Xl "^oJ' 

21 \0 J ' 22 \0 s 2 ) ' 2 \0j ' 
Yx= (0 yi),y 2 = (0 y 2 ). 

Since si, s 2 are eigenvalues, we have s\ — s 2 — 1 and det .Dj = z, which 
gives z — 1. Now, by solving the equations BiDiBi — DiBiDi = and 
B i+ iDiB i+1 - DiB i+1 Di = we obtain t x =t 2 = 1, t>it> 2 = 1, y 2 = 
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X 2 = X!V 2 , Xiyi 

(h{i-x) 

1 1 

1 

a: 1 





V Vi 



0. Thus, for % — 1, 



A = 



...,r 







^2( ff -i-l) 





1 we have 


o ^ 















1 







o 




1 J 





= 0. 





ccj 



1 1 
1 


OLiVi 

Similarly, using the relations between D r and A iy B t it can be shown 
that 

(hg-2 \ 
I I X r 

10 
y r 1 / 

It is not possible that x r = y r = 0, because then D r = A r and Lemma 
3.4 would give a contradiction. For 1 < i < r — 1, by solving the 
equation DiD r — D r Di = we obtain Xiy r = and x r y.i = 0. It follows 
that either Xi = for all % = 1, . . . , r, or yi = for all i = 1, . . . , r. 
We are going to show that it is possible to change the basis so that 
ctj = — 1 for % — 1, . . . , r — 1 and x r + y r = —2. Suppose that the old 
basis is @i = (vi, w±, . . . ,v r , w r , v r+1 ). We consider two cases. 
Case 1: x r = 0. Then y r ^ and the new basis is: 



D r 



\ 



v i = (-l) r % OLi ■ ■ ■ a r _iVi, Wi = (-l) r % ai 



1. 



r - 1 



~2 Vr+1 - 



J ri u r +l 

In the new basis we have: 

D r = Vi(tg 2r+1 ), D l = C t + x\ (E 2r+1>2i - E, 

for i — 1 , . . . , r — 1 . 

Case 2: y r = 0. Then x r 7^ and the new basis is: 



2r+l,2i+2 



), 



,r-l,V r = ~Y V n W r z 



r—i+1 



-■uv 



i = 1 

In the new basis we have: 

D r = ^2(ts 2r+1 ), Di = Ci + x\ (E 2i _ lj2r+1 — E 2i+1:2r+1 ) , 

for i — 1, . . . ,r — 1. 

Since U 2r+ i commutes with A4 and Bi for 1 < i < r — 1, thus, by 
Lemma 2.1, 

C/ 2r +i = diag (Ai/ 2 , A2/2, • • • , A r _i/2, X) , 
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for some 3x3 matrix X. The relations A r U 2r +i = U 2r+i A r (R8) and 
D r U 2r+1 D r = U 2r+1 (R12) imply that X has the form 

(A r a X r \ / \ r a f3 \ 

X r or X = A r 
/3 -\ r j \0 X r -X r ) 

respectively in case 1 and case 2. For 1 < % < r — 1, by the relation (R6) 
we have DiU 2r +i — U 2r +±Di = 0. By solving this equation we obtain 
Xi = Xi + \ and x\ = 0, hence Di = Ci. We also see that U 2r +i has two 
eigenvalues A r ,— A r with #A r = 2r. Since U 2r+ i is conjugate to U 2l } +1 
we have A r G { — 1, 1} and by multiplying / by (— l) ab if necessary, we 
may assume A r = 1. 

By the relation (Rll) we have 

U 2r = (B r C r ) 1 ?7 2r 1 fl (-B r C , r ), 

U 2r —\ = (yB r C> * r C < r —\B r ) ^U 2r -\-\ (^B r C r C^ — i-Bj-) ■ 

Similarly as in the proof for odd g, by solving U 2r +\U 2r -\— U 2r -iU 2r+1 = 
we obtain (3 = —2a, and then by solving U 2r +iU 2r U 2r +i— U 2j -U 2r+ iU 2r = 
we obtain a = — 1 in the case 1, or a — 1 in the case 2. Hence 
U 2r +i = ^i(u 2r+ i) in the case 1, or U 2r +\ = ^ 2 {u 2r+ i) in the case 2. 
By Theorem 3.1, / is equal to in the case 1, and equal to ^ 2 in the 
case 2, on generators of M(N). □ 

8. Homomorphisms from M(N 8 ) to GL(7, C) 

The aim of this section is to prove Theorem 1.6. First we have to 
define the epimorphism e: A^(A^ 2r +2) — > Sp(2r, Z 2 ). 

Fix r > 1 and set V = H 1 (N 2r+2: Z 2 ). V is a vector space over 
Z 2 of dimension 2r + 2 with basis xi = [£J 2 for 1 < i < 2r + 2, 
where [£j] 2 denotes the mod 2 homology class of the curve £j. The mod 
2 intersection pairing is the symmetric bilinear form on V satisfying 
{xi,~x]} 2 = 5ij. We define another basis for V. For 1 < i < r we set 

Vi = [ei} 2 = x~[ -\ hxi7, Wi = [£m]2 = x^ + ^m, 

c = x 2r _|_ 2 , (i = Xi + • • • + x 2r + 2 . 

Let Iso(V) denote the group of automorphisms of V preserving (•, -) 2 . 

Lemma 8.1. The group Iso(^) is isomorphic to a semi- direct product 
Sp(2r,Z 2 ) X Zf +1 . 

Proof. It is easy to check that d is the unique vector of V satisfying 
(x,d) 2 = (x,x) 2 for all a; G V, which implies that d is fixed by all 
elements of lso(V). 
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Let W = span{fj,Wj \ i = 1, . . . ,r} and observe that the restriction 
of (•, -) 2 to W is nondegenerate and (x, x) 2 = for x G W, hence it is 
a symplectic form on W. For R G Sp(W) we define A R G Iso(V) as 

A R (d) = d, A R (c) = c, A R (x) = R(x) for x G W. 

It is easy to check that W = {x G V \ (x, d) 2 = (x, c) 2 = 0}. It follows 
that if L G Iso(V) fixes c, then since L(d) = d, L preserves W, and 
hence L = A R for some R G Sp(W). Thus the mapping R h-> A R 
defines an isomorphism Sp(VF) — > Stabwy) (c) . 
For x G Z 2 and z G W we define ^ G Iso(l / ) as 

B x ,z(d) = d, B x ^ z (c) = c+xd+z, B XjZ (w) = w+(w, z) 2 d for w 6 W. 

Let 

iV = {B x z | a; G Z 2 ,zG W^}. 
This is a subgroup of Iso(V) with the group law 

B XI:ZI B X2Z2 = B xl + X2 +( zl: z 2 } 2iZl + z , 2 . 

It follows that N is abelian and B 2 X Z = 1 for all x, z. Thus TV is 
isomorphic to Z 2 r+1 . 

Let L G Iso(V) be arbitrary. Since (L(c),d) = (L(c),L(d)) = 
(c, d) = 1, thus L(c) = c + xd + z for some x G Z 2 , z G W 7 . It follows 
that B~\L G Stabi so (y)(c) and hence L = ^^Ar for some R G Sp(W). 
This decomposition is clearly unique, and since AuBx^A^ 1 = B XiR f z \, 
thus N is normal in Iso(V) and lso(V) = N xi Stabi so (y)(c). □ 

Lemma 8.2. For r > 2 i/iere is an epimorphism 

e: M{N 2r+2 ) -> Sp(2r,Z 2 ), 

whose kernel is normally generated by ts 2r+1 U2 r +i and ts 2r+1 t~\ 

Proof. Let A4 = M{N 2r+ 2)- The action of on V = H 1 (N 2r+ 2, Z 2 ) 
induces a homomorphism p: M. ^ Iso(V r ), which was proved to be 
surjective in [8] and [17], and whose kernel is the normal closure of 
ts 2r+1 u 2r +i by [23]. By Lemma 8.1, there exists a normal subgroup ./V" 
of Iso(V), such that Iso(V)/iV is isomorphic to Sp(2r, Z 2 ). We define 
e to be the composition of p with the canonical projection Iso(V) — >■ 
lso(V)/N. 

Let K be the normal closure of ts 2r+1 u 2r +i and ts 2r+1 t~ r 1 in Ai. We 
claim that K C kere. We have t5 2r+1 -U2r+i G kerp C kere. For 
x G V we have p(t £r )(x) = x + {v r ,x) 2 v r and p(4 2r+1 )(^) = x + 
([^+1)2,^) [^2r+i]2- Since [^+1)2 — v r + d, it is not difficult to check 
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thatp(t,5 2r+1 ) = B ltVr op(t er ), which gives p(t 52r+1 t £r 1 ) G iV and t &2r+1 t £ ^ G 
ker e. It follows that there is an induced epimorphism 

e': M/if -> Iso(y)/JV = Sp(2r,Z 2 ). 

To prove that e' is an isomorphism, it suffices to show [M. : if] < 
|Sp(2r, Z 2 )|. We are going to prove the last inequality by exhibiting an 
epimorphism Sp(2r, Z 2 ) — >■ M./K. 

Observe that the map 77: Ai(S') — >■ M./K defined to be the com- 
position of t: M(S') — >■ from Corollary 3.6 with the canonical 
projection 7r: .M — >■ M./K is surjective, because .M is generated by 
twists about curves on P(S') and tg 2r+l u 2r +i by Theorem 3.1. Glu- 
ing a disc along the boundary component of S' bounding a pair of 
pants with a r and 7 r induces an epimomorphism Ai(S') — > Ai(S r> i) 
whose kernel is normally generated by t lr t~^ (see [15, Proposition 
3.8]). Since i{t lr t~^) = ts 2r+l t~ 1 G K, it follows that we have an in- 
duced epimorphism rj' : A4(S rj i) — > M./K. There is an epimorphism 
•M-{S r ,\) — > Sp(2r, Z 2 ) induced by the action of .M (SV,i) on Hi(S rt ±, Z 2 ), 
whose kernel is normally generated by t 2 ai (see [2, Theorem 5.7], here 
we are using the assumption r > 2). By applying Lemma 3.4 (with 
i = r , j = 2r + 1) to tt: M ->■ .M/if, we have r/(^J = ^(^J = 1- It 
follows that there is an induced epimorphism rj" : Sp(2r, Z 2 ) — >■ M./K. 

M(S') -5— >• M — ^ M/K 




Sp(2r, Z 2 ) 



The existence of rj" proves that e' is an isomorphism and K = ker e. □ 

Lemma 8.3. Suppose that f: A4(N 8 ) — > GL(7, C) is a homomor- 
phism, such that /(^J has order 2. Then f or (— l) ab / factors through 
the epimorphism e: Ai(Ns) — > Sp(6,Z 2 ). 

Proof. Let H be the normal closure of tg in M. = Ai(N 8 ) and G = 
Ai/H. Since H C ker/, we have a homomorphism /': G — > GL(7, C) 
such that f = /' o tt, where 7r: — >■ G is the canonical projection. 
There is a homomorphism p: & 8 — > G, defined as p(<7j) = ^(t^J, where 
o~i = (i,i + l), for 1 < i < 7. Let 0: ©8 — >■ GL(7, C) be the composition 
= /' P- If is reducible, then lm(0) is abelian by Lemma 2.2, 
f(ttii) — <K°i) — 0(°"2) = f(t$ 2 ), and Im(/) is also abelian by Lemma 
3.3, which implies f(t Sl ) = 1 by Theorem 3.2, a contradiction. Hence 
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is irreducible and since det/^J = 1 (by Theorem 3.2), is the 
tensor product of the standard and sign representations (by Lemma 
2.2). For 1 < % < 7 set L, L = f(t 5i ) = 0(crj). With respect to some basis 
(v i, . . . , v 7 ) we have 

Li = diag (A, -I 5 ) , L 7 = diag (-J 5 , B) , Li = diag (-h-2, C, -7 6 -i) 
for 2 < i < 6, where 

Let M be the matrix of fie^). Since M commutes with Lj for i 7^ 6 
(R5), it preserves E(Li, 1) = span{t>j}. Hence M(vi) = XiVi for i 7^ 6 
and M(vq) = y\V\ + • • • + 2/7^7, for some complex numbers Xi,yj. By 
solving the equations MLi = LiM for 1 < % < 5 and 2 = 7 we obtain 

£i = xi, y» = iyi for 1 < i < 5, y 6 = xi + 6yi, z 7 = y 6 - 2y 7 . 

Since M and Lj are conjugate, they have the same eigenvalues, which 
gives Xi = — 1 and y Q = —x 7 . If y§ = 1, then yi = 1/3, y 7 = 1, which 
contradicts the braid relation ML 6 M = LqMLq (R5). Hence y6 — ~~ L 
l/i = 0, 2/7 = —1, which means M = L 7 . 

For % — 1, . . . 7 let C/j be the matrix of f{uj). Since C/ 7 commutes 
with Lj for 1 < j < 5 (R6) and with M = L 7 (R8), we obtain, as 
above, that 

U-jivi) = xvi for 1 < i < 5, 

^7(^6) = 2/(^1 + 2w 2 + 3w 3 + 4v A + 5v 5 ) + (x + 6y)v 6 + zv 7 
U 7 (v 7 ) = (x + Qy - 2z)v 7 

for some complex numbers x,y,z. Since U 7 is conjugate to its inverse, 
and x is an eigenvalue of multiplicity at least 5, thus x — ±1, and 
by multiplying / by (— l) ab if necessary, we may assume x = —1. 
By (Rll) we have U5 = (LqL 7 L 5 Lq)^ 1 U 7 (LqL 7 L 5 Lq) and by solving 
^5^7 — U 7 U$ we obtain y = 0. Since det U 7 = ±1, either — 1 — 2z = 1 
or — 1 — 2z = — 1. In the latter case we have C/7 = —J, and since U 6 
is conjugate to U 7 , thus C/ 6 = —J, and the relation L 6 U 7 Uq = U 7 UqL 7 
(RIO) gives L 6 = L 7 , a contradiction. Hence z — — 1 and C/ 7 = L 7 . 

We have M = U 7 = L 7 and since L 7 = /, thus {ts 7 tj^, t$ T u 7 } C ker /, 
which implies, by Lemma 8.2, that / factors through e. □ 

Proof of Theorem 1.6. Suppose that /: Ai(Ng) — > GL(7, C) is a homo- 
morphism, such that Im(/) is not abelian. By Lemma 7.2, L = /(^J. 
has an eigenvalue A such that dim E(L, A) = 6. Since L is conjugate 
to L^ 1 we have A 2 = I. Suppose that A = —1. Then since detL = 1 
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we have #A = 6, and there is another eigenvalue \i= \. It follows that 
L has order 2 and the case (2) holds by Lemma 8.3. If A = 1 then it 
must be the unique eigenvalue, and the case (3) holds by Lemma 7.3 
and the proof of Theorem 1.5 for even g. □ 

Remark 8.4. Suppose that G is a finite quotient of Ai(N g ) for g > 7, 
g ^ 8, and /: G — > GL(g — 1,C) is a homomorphism. Then, by 
Theorem 1.5, Im(/) is abelian, and if G is perfect, then / must be 
trivial. For example, by Lemma 8.2, for r > 4, the only homomorphism 
from Sp(2r, Z 2 ) to GL(2r + 1, C) is the trivial one. 
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